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1 Introduction 

Wirtinger's calculus [15] has become very popular in the signal processing community mainly in the context 
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pq . of complex adaptive filtering [131 E [D El [I2l El HI [lO] , as a means of computing, in an elegant way, gradients of 

real valued cost functions defined on complex domains (C^). Such functions, obviously, are not holomorphic 

d ' and therefore the complex derivative cannot be used. Instead, if we consider that the cost function is defined 

on a Euclidean domain with a double dimensionality {M?'^), then the real derivatives may be employed. The 

price of this approach is that the computations become cumbersome and tedious. Wirtinger's calculus 

^SJ ' provides an alternative equivalent formulation, that is based on simple rules and principles and which bears 

a great resemblance to the rules of the standard complex derivative. Although this methodology is relatively 

r*n , known in the German speaking countries and has been applied to practical applications [31 [5], only recently 

") I has become popular in the signal processing community, mostly due to the works of Picinbono on widely 

^ ■ linear estimation filters |13j . 

O . Most Complex Analysis' textbooks deal with complex analytic (i.e., holomorphic) functions and their 

properties, which in order to be studied properly a great deal of notions from topology, differential geometry, 

calculus on manifolds and from other mathematical fields need to be employed. Therefore, most of these 

materials are accessible only to the specialist. It is only natural that in the scope of this literature, Wirtinger's 

[^ I calculus is usually ignored, since it involves non-holomorphic functions. Nevertheless, in some of these 

textbooks, the ideas of Wirtinger's calculus are mentioned, although, in most cases, they are presented 

either superficially, or they are introduced in a completely different set-up (mainly as a bi-product of the 

^iD '. Cauchy Riemann conditions). Some of these textbooks are [14^ \TT\ [9]. However, most of these works are 

^^ ] highly specialized and technically abstruse, and therefore not recommended for someone who wants only to 

understand the concepts of Wirtinger's calculus and use them in his/her field. Moreover, a rigorous and 

self-consistent presentation of the main ideas of Wirtinger's calculus cannot be found in any of those works. 

K^ \ An excellent and highly recommended first attempt to summarize all the related concepts and present them 

;^ ' in a unified framework is the introductory text of K. Kreutz-Delgado [6]. The aim of the present manuscript 

^ is twofold: a) it endeavors to provide a more rigorous presentation of the related material, focusing on 

aspects that the author finds more insightful and b) it extends the notions of Wirtinger's calculus on general 

Hilbert spaces (such as Reproducing Hilbert Kernel Spaces). 

A common misconception (usually done by beginners in the field) is that Wirtinger's calculus uses an 
alternative definition of derivatives and therefore results in different gradient rules in minimization problems. 
We should emphasize that the theoretical foundation of Wirtinger's calculus is the common definition of 
the real derivative. However, it turns out that when the complex structure is taken into account, the real 
derivatives may be described using an equivalent and more elegant formulation which bears a surprising 
resemblance with the complex derivative. Therefore, simple rules may be derived and the computations of 
the gradients, which may become tedious if the double dimensional space M?'^ is considered, are simplified. 
The manuscript has two main parts. Section [21 deals with ordinary Wirtinger's calculus for functions of 
one complex variable, while in section [3] the main notions and results of the extended Wirtinger's Calculus 
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in RKHSs are presented. Throughout the paper, we will denote the set of all integers, real and complex 
numbers by N, M and C respectively. Vector or matrix valued quantities appear in boldfaced symbols. 

The present report, has been inspired by the need of the author and its colleagues to understand the 
underlying theory of Wirtinger's Calculus and to further extend it to include the kernel case. Many parts 
have been considerably improved after long discussions with prof. S. Theodoridis and L. Dalla. 

2 Wirtinger's Calculus on C 

Consider the function / : X C C — > C, f{z) = f{x + iy) = u{x, y) + v{x, y)i, where n, and v are real valued 
functions defined on an open subset X of M^. Any such function, /, may be regarded as defined either on a 
subset of C or on a subset of M^. Furthermore, / may be regarded either as a complex valued function, or 
as a vector valued function, which takes values in M?. Therefore, we may equivalently write: 

f{z) = f{x + iy) = f{x, y) = u{x, y) + iv{x, y) = {u{x, y),v{x, y)). 

The complex derivative of / at c, if it exists, is defined as the limit: 

■' ^ ' h^o h 

This definition, although similar with the typical real derivative of elementary calculus, exploits the complex 
structure of X. More specifically, the division that appears in the definition is based on the complex 
multiplication, which forces a great deal of structure on /. From this simple fact follow all the important 
strong properties of the complex derivative, which do not have counterparts in the ordinary real calculus. 
For example, it is well known that if /' exists, then so does /'"\ for n S N. If / is differentiable at any 
zq (^ A, f is called holomorphic in A, or complex analytic in A, in the sense that it can be expanded as a 
Taylor series, i.e., 

n- + h) = Y^L_^h\ (1) 

n=0 

The proof of this statement is out of the scope of this manuscript, but it can be found at any complex 
analysis textbook. The expression "/ is complex analytic at zq" nieans that / is complex-analytic at a 
neighborhood around zq. We will say that / is real analytic, when both u and v have a Taylor's series 
expansion in the real domain. 

2.1 Cauchy-Riemann conditions 

We begin our study, exploring the relations between the complex derivative and the real derivatives. In the 
following we will say that / is differentiable in the complex sense, if the complex derivative exists, and that 
/ is differentiable in the real sense, if both u and v have partial derivatives. 

Proposition 2.1. If the complex derivative of f at a point c (i.e.,f'{c)) exists, thenu andv are differentiable 
at the point (ci, C2), where c = ci + C2i. Furthermore, 

uu uv ou uv 

o-(ci>C2) = ^(ci,C2) and — (ci,C2) = -^(ci,C2). (2) 

There are several proves of this proposition, that can be found in any complex analysis textbook. Here we 
present the two most characteristic ones. 



1st Proof. Since / is differentiable in the complex sense, the hmit: 

f{c + h)-f{c) 



f'{c) = hm 



/i-)-o h 

exists. Consider the special case where h = hi (i.e., h ^ on the real axes). Then 

f{c + h) - f{c) _ u{ci + /ii, C2) + ivjci + /ii, C2) - -u(ci, C2) - iv{ci,C2) 
h hi 

_ U(C1 + hl,C2) -U{CI,C2) .V(CI +hl,C2) -v{ci,C2) 



hi hi 

In this case, since the left part of the equation converges to /(c), the real and imaginary parts of the second 
half of the equation must also be convergent. Thus, u, and v have partial derivatives with respect to x and 
/'(c) = du{c)/dx + idv{c)/dx. Following the same rationale, if we set h = ih2 (i.e., /z — )■ on the imaginary 
axes), we take 

fjc + h) - f{c) _ m(ci,C2 + /j-2) + ivici,C2 + h2) -u{ci,C2) -iv{ci,C2) 
h i/i2 

_ u{ci,C2 + h2) -u{ci,C2) . v{ci,C2 + h2) --u(ci,C2) 
ih2 «/i2 

_ ■^(ci,C2 + /t2) -''^(ci,C2) _ . -u(ci,C2 + /?-2) -^(ci,C2) 
/l2 h2 

The last equation guarantees the existence of the partial derivatives of u and v with respect to y. We 
conclude that u and v have partial derivatives and that 

D 

2nd Proof. Considering the first order Taylor expansion of / around c, we take: 

f{c + h) = f{c) + nc)h + o{\h\), (3) 

where the notation o means that o(|/i|)/|/i| — )• 0, as \h\ — >■ 0. Substituting /'(c) = A + Bi we have: 

/(c + h) =f{c) + {A + Bi){hi + ih2) + o{h) 

='u(ci, C2) + Ahi - Bh2 + K[o(/i)] + i (v(ci, C2) + Bhi + Ah2 + 9=[o(|/i|)]) . 



Therefore, 



u(ci + /ii,C2 + /i2) =u{ci,C2) + Ahi -Bh2 + '^[o{\h\)], (4) 

v{ci + hi,C2 + h2) =v{ci,C2)+Bhi+Ah2 + %o{\h\)]. (5) 



Since o(|/i|)/|/i| — )• 0, we also have 



3f?[o(|(/ii,/i2)|)]/y /i? + /li ^ and 9[o(|(/ii, /i2)|)]/\//i? + /i^ ^ as /i ^ 0. 

Thus, equations ([DlS]) are the first order Taylor expansions of u and v around (ci,C2). Hence we deduce 
that: 

UU OU UV ov 

^(C1,C2) = A,-{CI,C2) = -B,g^{ci,C2) = B,g^{ci,C2) = A. 

This completes the proof. D 



Equations ([2]) are called the Cauchy Riemann conditions. It is well known that they provide a necessary 
and sufficient condition, for a complex function / to be differentiable in the complex sense, providing that 
/ is differentiable in the real sense. This is explored in the following proposition. 

Proposition 2.2. /// is differentiable in the real sense at a point (ci,C2) and the Cauchy- Riemann condi- 
tions hold: 

du dv du dv . 

^(ci, C2) = ^(ci, C2) and ^(ci, cs) = -^(ci, C2), (6) 

then f is differentiable in the complex sense at the point c = ci + C2i- 



Proof. Consider the first order Taylor expansions of u and u at c = ci + ZC2 = (ci, C2): 

uu uu 

u{c + h) = u{c) + ■7ric)hi + TT (c)^2 + o{\h\), 
ox oy 

dv dv 

v{c + h) = v{c) + -Q^{c)hi + ^(c)/i2 + o{\h\). 

Multiplying the second relation by i and adding it to the first one, we take: 

/(c + h) = f{c) + (|^(c) + .|^(c)) h, + (|^(c) + z|(c)) h, + 0{\h\). 

To simplify the notation we may define 

|i(., = |!i(.) + i|!;(c) and §^(c) = ^W + i^^M 
OX ox ox oy oy oy 

f{c +h) = /(C) + ^ic)h + ^ic)h2 + 0{\h\). 

Nest, we substitute hi and /i2 using the relations hi = -^ — and /12 = 2t • 

/(o + h) = m + i (|(c) + i| w) k + 1 (|(c) - 1|m) k- + „(i.i) 

It will be shown that equation d?]) is essential for the development of Wirtinger's calculus. To complete the 
proof of the proposition we compute the fraction that appears in the definition of the complex derivative: 

/(c + /i)-/(c) 1/5/,, .9/, A Ifdf,, .df,,\h* o{\h\) 



and obtain: 



h 2 \dx dy J 2 \dx dy J h h 

Recall that for the limit lim^.^^o -^x ^° ^xist, it is necessary that A = Clj. Hence, since o{\h\)/h ^^ as 
/i — >■ 0, / is differentiable in the complex sense, iff 



df,, df , , 



However, according to our definition. 



S(^)-'|(=)K£<^>-I''')^<£<='^l<^"- 



^To prove it, just set h = re'" and let r ^> 0, while keeping 6 constant. Then A^ — Xe ^'" — >■ 0, if and only if A = 



Thus, / is differentiable in the complex sense, iff the Cauchy-Rieniann conditions hold. Moreover, in this 
case: 






1 f du , . dv , .\ i f du , . dv , . 



2 \dx dx J 2 \dy dy 

1 f du , . dv , .\ i / dv , . du , . 



2 \dx dy J 2 \dx dy 

du . . .dv . . 

-d^^'^^'d^^'^ 
dv du 

'-d^^'^-%^'^- 



n 



More information on holomorphic functions and their relation to harmonic real functions may be found 
in [I1[II1[9]. 

2.2 Wirtinger's Derivatives 

In many practical applications we are dealing with functions / that are not differentiable in the complex 
sense. For example, in minimization problems the cost function is real valued and thus cannot be complex- 
differentiable at every x € X (unless it is a constant functioro)- In these cases, our only option is to 
work with the real derivatives of u and v. However, this might make the computations of the gradients 
cumbersome and tedious. To cope with this problem, we will develop an alternative formulation which, 
although it is based on the real derivatives, it strongly resembles the notion of the complex derivative. In 
fact, if / is differentiable in the complex sense, the developed derivatives will coincide with the complex 
ones. To provide some more insights into the ideas that lie behind the derivation of Wirtinger's Calculus, we 
present an alternative definition of a complex derivative, which we call the conjugate- complex derivative at c. 
We shall say that / is conjugate- complex differentiable (or that it is differentiable in the conjugate- complex 
sense) at c, if the limit 

■'*^ ' h*-,o h h^o h* ^ ' 

exists. Following a procedure similar to the one presented in section [2TT] we may prove the following propo- 
sitions. 

Proposition 2.3. If the conjugate- complex derivative of f at a point c (i.e.,fl{c)) exists, then u and v are 
differentiable at the point (01,02), where c = ci + C2i. Furthermore, 

du dv du dv 

^(ci,C2) = -^(ci,C2) and ^(ci.ca) = ^(ci,C2). (9) 

These are called the conjugate Cauchy Riemann conditions. 

Proposition 2.4. If f is differentiable in the real sense at a point (ci,C2) and the conjugate Cauchy- 
Riemann conditions hold, then f is differentiable in the conjugate- complex sense at the point c = ci + C2i. 

If a function / is differentiable in the conjugate-complex sense, at every point of an open set A, we will say 
that / is conjugate holomorphic on A. As in the case of the holomorphic functions, one may prove that 



^This statement can be proved using the Cauchy Riemann conditions. For any real valued complex function / defined on a 
, II vanishes. Therefore the Cauchy-Riemann conditions dictate that du/dx — du/dy — 0. Hence, u must be a constant. 



similar strong results hold for conjugate-holoniorphic functions. In particular, it can be shown that if fi{z) 
exists for every z in a neighborhood of c, then / has a form of a Taylor series expansion around c, i.e., 

fic + h) = Y,^^{hT. (10) 

n=0 

In this case, we will say that / is conjugate- complex analytic at c. Note, that if f{z) is complex analytic at 
c, then f*{z) is conjugate-complex analytic at c. 

It is evident that if neither the Cauchy Riemann conditions, nor the conjugate Cauchy-Riemann con- 
ditions are satisfied for a function /, then the complex derivatives cannot be exploited and the function 
cannot be expressed neither in terms of h or h* , as in the case of complex or conjugate-complex differen- 
tiable functions. Nevertheless, if / is differentiable in the real sense (i.e., u and v have partial derivatives), 
we may still find a form of Taylor's series expansion. Recall, for example, that in the proof of proposition 
12. 2^ we concluded, based on the first order Taylor's series expansion of n, v, that (equation d?])): 

/(c + k) = /(c) + 1 (|(c) - ,|(e)) /. + i (I (c) + .|(c)) k- + „(|.|). 

One may notice that in the more general case, where / is real-differentiable, it's Taylor's expansion is casted 
in terms of both h and h* . This can be generalized for higher order Taylor's expansion formulas following 
the same rationale. Observe also that, if / is complex or conjugate-complex differentiable, this relation 
degenerates (due to the Cauchy Riemann conditions) to the respective Taylor's expansion formula (i.e., ([T]) 
or nop). In this context, the following definitions come naturally. 

We define the Wirtinger's derivative (or W-derivative for short) of / at c as follows 

|{(^) - \ ifj'^ - f<')) - \ (If <=) - !<=)) - \ &^> - S<^)) ■ <") 

Consequently, the conjugate Wirtinger's derivative (or CW-derivative for short) of / at c is defined by: 

l^(=) 4 (S<') - f <')) - 5 (If <=) - 1(=)) - \ (If (=) - !<')) ■ <-) 

Note that both the W-derivative and the CW-derivative exist, if / is differentiable in the real sense. In view 
of these new definitions, equation ([7]) may now be recasted as follows 

/(c + h) = fie) + ^ic)h + ^{c)h* + oi\h\). (13) 

At first glance the definitions the W and CW derivatives seem rather obscure. Although, it is evident that 
they are defined so that that they are consistent with the Taylor's formula (equation ([7])), their computation 
seems quite difficult. However, this is not the case. We will show that they may be computed quickly 
using well-known differentiation rules. First, observe that if / satisfies the Cauchy Riemann conditions then 
the W-derivative degenerates to the standard complex derivative. The following theorem establishes the 
fundamental property of W and CW derivatives. Its proof is rather obvious. 

Theorem 2.5. /// is complex differentiable at c, then its W derivative degenerates to the standard complex 
derivative, while its CW derivative vanishes, i.e., 

^f I \ f'f \ ^f ( \ n 
gj(c) = /(c), ^^-)=^- 

Consequently, if f is conjugate- complex differentiable ate, then its CW derivative degenerates to the standard 
conjugate- complex derivative, while its W derivative vanishes, i.e., 

^(c) = A(c), ^(c)=0. 
6 



In the sequel, we will develop the main differentiation rules of Wirtinger's derivatives. Most of the proofs 
of the following properties are straightforward. Nevertheless, we present them all for completeness. 

Proposition 2.6. // / is differentiable in the real sense at c, then 



Proof. 






2 \dx dy J 2 \ dx dy 

df* 



n 



Proposition 2.7. // / is differentiable in the real sense at c, then 






Proof. 



^/ / n\* 1 fdu. . dv . .\ i f dv du . . 



D 



Proposition 2.8 (Linearity). If f , g are differentiable in the real sense at c and a,/3 G C, then 



d{af + Pg) df dg 

^^^(c) = a-(c)+/3-(c), (16) 

^5;^(c) = a^(c) + /3^(c) (17) 



Proof. Let f{z) = f{x,y) = Uf{x,y) + ivf{x,y), g{z) = g{x,y) = Ug{x,y) + ivg{x,y) be two complex 
functions and a, f3 € C, such that a = ai + ia2, /5 = /3i + i/?2. Then 

r{z) =af{z) + j3g{z) = (oi + ia2){uf{x, y) + ivf{x, y)) + (^i + i/32){ug{x, y) + ivg{x, y)) 
= {aiUf{x, y) - a2Vf{x, y) + /3iUg{x, y) - l32Vg{x, y)) 
+ i{aiVf{x,y) + a2Uf{x,y) + f3iVg{x,y) + (32Ug{x,y)) . 



Thus, the W-derivative of r will be given by: 

Or , , 1 / dur , . dvr , C\ if dvj. , , dur , . 



|f(^)-'''|f(^)-''^l?<=) 



= 2 («1 + ^«2)^(C) + -(«!+ ^«2)^(C) + -(/3l + iP2)-Q^{c) + 2 (/?! + ^/52)^(C) 

+ 2(«i + ^"2)^(c) - -(ai + ^«2)^(c) + 2 (/3i + ^/52)^(c) - 2(/5i + ^/52)^(c) 
1 f duf ^^/ / \\ ^ [ ^'^f t ^^/ 



On the other hand, in view of Propositions 12.71 and 12.61 and the linearity property of the W-derivative, the 
CW-derivative of r at c will be given by: 

dr ^ ^ d{af + fig)^^ ( d{af + pg)* 



d{a* f* + p* g*) , X f *df\ , n*dg% , 



dz J \ dz dz 

(^f*(^\\R(^9*X df dg 



n 



Proposition 2.9 (Product Rule). If f , g are dijferentiable in the real sense at c, then 

g^(c)^^(c).(.)./(c)^(c). 0^ 

Proof. Let f{z) = f{x,y) = Uf{x,y) + ivf{x,y), g{z) = g{x,y) = Ug{x,y) + ivg{x,y) be two complex 
functions differentiable at c. Consider the complex function r defined as r[z) = f{z)g{z). Then 

r{z) = {Uf{z) + iVg{z)){Ug{z) + iVg{z)) = [UjUg - VfVg) + i{UfVg + VfUg). 

Hence the W-derivative of r at c is given by: 

dr I ( dur . . dvr , S\ i ( dvr , . dur 

9^^^) =2 [-d^^'^ + 9^(^V + 2 ( 5^(^) - ^(^) 

1 fd{UfUg-VfVg) ^ djUfVg+VfUg) \ _^i_ f OJU fV g + V f Ug) djUfUg-VfVg) 



2 \ dx dy J 2 \ dx dy 

1 f djUfUg) _ djVfVg) ^ djUfVg) ^ djVfUg) 

2 \ dx dx dy dy 

+ 1 (^(^(c) + ^^ic) - ^%^(c) + ^(c: 

2 \ dx dx dy dy 



1 / duf dug dvf dv, 






5u 



W%W - ^W«/W + ^(^)^5(^) + ^(^)"/(^)) ■ 



5y 
After factorization we obtain: 



Applying the simple rule 1/i = —i, we take: 



which gives the result. 

The product rule of the C W-derivative follows from the product rule of the W-derivative and Propositions 
EJl O as follows: 

difg),, fdifg)*.X f9f*..,.. dg%.^,.. 

D 
Lemma 2.10 (Reciprocal Rule). /// is dijferentiable in the real sense at c and f{c) ^ 0, then 

^(0 - -Mi^ (21) 

8z- '"' /2(c) ■ * ' 

Proof. Let f{z) = Uf{x, y) + ivf{x, y) be a complex function, differentiable in the real sense at c, such that 
/(c) 7^ 0. Consider the function r{z) = 1/ f{z). Then 

Uf{z) . t;/(z) 



r{z) 



uj{z) + vj{z) uj{z) + vj{z) 



For the partial derivatives of Ur, Vr we have: 

dur. J-^{c){u){c) + vj{c)) - 2u}{cf-^{c) - 2uf{c)vf{cf^{c) 



dur, , J-^ic){u}{c) + vj{c)) - 2n}{cf-^{c) - 2uf{c)vf{cf-§^{c) 



-(c) 



dy" {u){c) + v){c)Y 

dvr.._ '^ic){u}{c) + v}{c)) - 2vj{cf^{c) - 2uf{c)vf{cf-^{c) 
dx^"' [u){c) + v){c)Y 

dvr '-^{c){u}{c) + vjjc)) - 2v}{cf-^{c) - 2uf{c)vf{cf-^{c) 

dy^""' {u}{c)+vj{c)r 

Therefore, 

dr 1 fdur. . dvr , A , i f dvr , . dur . 

= 2{u''{c)\vHc)Y (?^^"^ ("""'^"^ + ""'^"^ + 2iuf{c)vf{c)) + ^(c) {-2uf{c)vf{c) - iu) + iv){c)) 

+^(c) (-w/(c) + v}{c) + 2in/(c)r;/(c)) + ^(c) (2u/(c)^;/(c) + ^(c) - iv){c)) 

_ u){c)-v){c)-2iuf{c)vf{c) /_ /5^ 5^^ \ _ , fdv^ _ du^ 

2{u}{c)+vj{c)y \ \dx ^"^^ + «- ^^' ^ ^- ^"^^ ^- ^^' 

_ (92 



2(u^(c) + 'L'?(c))2 \ \ dx dy J \ dx dy 

(c) 



Pic)- 

To prove the corresponding rule of the CW-derivative we apply the reciprocal rule of the W-derivative 
as well as Propositions 12. 6^ 12.71 



Proposition 2.11 (Division Rule). ///, g are dijjerentiable in the real sense at c and g{c) ^ 0, then 

^(,).I(fM£W(£)|lM, p.) 

dz g\c) 

oz g [c) 

Proof. It follows immediately from the multiplication rule and the reciprocal rule ( ^^ = f{c) ■ —t-y I . D 

Proposition 2.12 (Chain Rule). // / is differentiahle in the real sense at c and g is dijjerentiable in the 
real sense at f{c), then 

10 



Proof. Consider the function h{z) = g o f[z) = Ug{uf{x,y),Vf{x,y)) + iVg{uf{x,y),Vf{x,y)). Then the 
partial derivatives of Uh and v^ are given by the chain rule: 

duh , , du„ , ^ , ..duf, , du„ , ^ , ,,dvf , . 

^(c) = |^(/(c))^(c) + |ii(/(c))|i(c). 
ay ox oy ay oy 

In addition, we have: 

|»(/(c))|i(c) =1 ('^(m)'^(c) + ^(/ W)|l W + i^(/(.))|i(c) - i|i(/(c))^(c) 

oz az 4 \ ox ox ox oy ox ox ox oy 

- If' ^<^>'^<^> - If' ^'=»l7 '^' - 't '^<^« t <^' - ^ t '^<^>' t '^' 

- '^'/<^))^'') - '^</''))|f' ') - ^'/<=»l^'') - ^'/'^))|f' ^) 

-1?</<=»l'<=> - 'If </<=))|f' ^' - ^'^<='' t '=' - '-^^'^<^^' 

and 

^</'^»f '^) 4 (^'/'=»l^'^) - ^'/'=»|f' =' - '^'^'^x t '^) - '^</'^)> t '^' 
-^|f<«'»l?"' - ^t <^''» t ''' - |f<^''» t "' - 1 '^<^>' t ''' 

Summing up the last two relations and eliminating the opposite terms, we obtain: 



|,;„)|„,^,;,„i^(,4(^,;,))^,)_.|. 



-'^'/<=»l:^<")-'^'/<=»?^<")-S'/<"))|f(=) 
-t</(^))t'='-t'^'=»l^'=' 



2 \ dx dy J 2 \ dx dy 

dh 

To prove the chain rule of the CW-derivative, we apply the chain rule of the W-derivative as well as 



11 



Propositions 12.61 12.71 and obtain: 

which completes the proof. D 

In the following we examine some examples in order to make the aforementioned rules more intelligible. 

1. f{z) = z^. Since / is complex analytic, Q^iz) = and -^{z) = 2z, for all z € C 

2. f{z) = z*. Since / is conjugate-complex analytic, -^{z) = and g^iz) = 1, for all z € C. 

3. f{z) = z^ -iz + {z*f. Applying the linearity rule we take: ^{z) = ^ + ^ + ^i^^ = Sz^ + i. 
Similarly, ^W = 2g>+ 111 + »<ga. 2,.^ 

4. f{z) = i. Applying the reciprocal rule: gj(-z) = — jj, gi^iz) = 0, for all 2 G C — {0}. 

5. f{z) = (z^ + z*) . Consider the functions (7(z) =z^ and h{z) = z'^+z*. Then/(z) = goh{z). Applying 
the chain rule: g(z) = ^,{{z^ + z*f)f^{z) + §,{{z^ + z*f)^{z) = 2,{z^ + z*f2z + Q = &z{z^ + z*f . 
Similarly, §i^{z) = ^[z'^ + z*f . 

It should have become clear by now, that all the aforementioned rules can be summarized to the following 
simple statements: 

• To compute the W-derivative of a function f , which is expressed in terms of z and z* , apply 
the known differentiation rules considering z* as a constant. 

• To compute the C W-derivative of a function f , which is expressed in terms of z and z* , 
apply the known differentiation rules considering z as a constant. 

Most texts or papers, that deal with Wirtinger's methodology, highlight the aforementioned rules only, 
disregarding the underlying rich mathematical body. Therefore, the first-time reader is left puzzled and 
with many unanswered questions. For example, it is difficult for the beginner to comprehend the notion 
"keep z* constant", since if z* is kept fixed, then so does z. We should emphasize, that these statements 
should be regarded as a simple computational trick, rather than as a rigorous mathematical rule. This 
trick works well, as shown in the examples, due to Theorem 12.51 which states the W and CW derivative 
vanish for conjugate-complex analytic and complex analytic functions respectively, and the differentiation 
rules (Propositions [2^2. 12p . Nonetheless, special care should be considered, whenever this trick is applied. 
Given the function f{z) = |zp, one might conclude that q^ = 0, since if we consider z as a constant, 
according to the aforementioned statements, then f{z) is also a constant. However, on a closer look, one 
may recast / as f{z) = zz*. Then the same trick produces g^ = z. Which one is correct? 

The answer, of course, is that q^ = z. One should not conclude that gp = 0, since / is not conjugate- 
holomorphic. We can only apply the aforementioned tricks on functions / that are expressed in terms of 
z and z* (hence the term f{z,z*) used often in the literature) in such a way, that: a) if we replace z with 
a fixed w, while keeping z* intact, the resulting function will be conjugate-complex analytic and b) if we 
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replace z* with a fixed w, while keeping z intact, the resulting function will be complex analytic. Utilizing 
the methodology employed in Proposition 12.21 one may prove that this is possible for any real analytic 
function /. Of course, another course of action (a more formal one) is to disregard the aforementioned tricks 
and use directly the differentiation rules (as seen in the examples). 

2.3 Second Order Taylor's expansion formula 

Let / a complex function differentiable in the real sense. Following the same procedure that led us to [7] we 
may derive Taylor's expansion formulas of any order. In this section we consider the second order expansion, 
since it is useful in many problems that employ Newton-based minimization. Consider the second order 
Taylor's expansion of u and v around c = ci + ic2'- 

uu uu 

U{c + h) = U{ci +hi,C2 + /l2) =U{CI,C2) + -K-{ci,C2)hl + ^(ci,C2)/l2 

+ ^{h^,h2)-H^-{hl,h2f + 0{\h\^), 

dv dv 

V{c+h) = V{CI +hi,C2 + /l2) =V{CI,C2) + T^(ci,C2)/ll + ^-(ci,C2)/l2 

OX oy 

+ ^{h^,h2) ■ H, ■ {hi,h2f + 0{\h\^), 

where Hu, H^ are the associated Hessian matrices. Since f{c + h) = u{c + h) + iv{c + h), after some algebra 
one obtains that: 

df df\ (h\ I /0(c) J^(c)\ (h 



f{c+h) = m + ^,-^ -L. +-{h,h*)- rf ' %n)'\-[u*]+o{\h\ 



2.4 Wirtinger's calculus applied on real valued functions 

Many problems of complex signal processing involve minimization problems of real valued cost functions 
defined on complex domains, i.e., / : X C C — )• M. Therefore, in order to successfully implement the 
associated minimization algorithms, the gradients of the respective cost functions need to be deployed. 
Since, such functions are neither complex analytic nor conjugate-complex analytiqfl, our only option is to 
compute the gradients, either by employing ordinary M^ calculus, that is regarding C as M^ and evaluating 
the gradient (i.e., the partial derivatives) of the cost function f{x + iy) = f{x,y), or by using Wirtinger's 
calculus. Both cases will eventually lead to the same results, but the application of Wirtinger's calculus 
provides a more elegant and comfortable alternative, especially if the cost function by its definition is given 
in terms of z and z* instead of x and y (i.e., the real and imaginary part of z). 

As the function under consideration f{z) is real valued, the W and CW derivatives are simplified, i.e., 

and the following important property can be derived. 

Lemma 2.13. If f : X QC ^ M. is differentiable in the real sense, then 

lf(c)Y = ^ic). (26) 

dz^ ^ dz*^ ' ^ ' 



^Of course, one may not use a complex valued function as a cost function of a minimization problem, since we cannot define 
inequalities in C 
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An important consequence is that if / is a real valued function defined on C, then its first order Taylor's 
expansion at z is given by: 

/(c + /i) = /(c) + |^(c)/i + ^(c)r + o(\h\) 

=/(c)+|{(c)/i+('|{(c)/.y+o(i/.j) 



/(c) + K 






+ o(|/.|). 



However, in view of the Cauchy Riemann inequality we have: 



3? 



dz 



{c)h 



3f? 



"■ ' fi(^) 



< 



^' ' |(^) 



<N 



5/ 

dz* 



(c) 



The equality in the above relationship holds, if /i fj -q^^. Hence, the direction of increase of / is -q^. 
Therefore, any gradient descent based algorithm minimizing f{z) is based on the update scheme: 



5/ 

dz* 



^n-l -/^- -^-^{Zn~l)- 



(27) 



Assuming differentiability of /, a standard result from elementary real calculus states that a necessary 
condition for a point {xo,yo) to be an optimum (in the sense that f{x,y) is minimized) is that this point 
is a stationary point of /, i.e. the partial derivatives of / at (xo,yo) vanish. In the context of Wirtinger's 
calculus we have the following obvious corresponding result. 

Proposition 2.14. // / : X C C — > M is differentiahle at x in the real sense, then a necessary condition 
for a point c to he a local optimum (in the sense that f{c) is minimized or maximized) is that either the W, 
or the CW derivative vanished. 



3 Wirtinger's Calculus on general complex Hilbert spaces 

In this section we generalize the main ideas and results of Wirtinger's calculus on general Hilbert spaces. 
To this end, we begin with a brief review of the Frechet derivative, which generalizes differentiability to 
abstract Banach spaces. 



3.1 Frechet Derivatives 

Consider a Hilbert space H over the field F (typically M or C). The operator T : H - 
Frechet differentiahle at /o, if there exists a linear continuous operator W = iWi, W2, ■ 
such that 



lim 



\T{fo + h)-T{fo)-Wih)\\ 



0, 



H 



F'^ is said to be 



(28) 



where || • \\h = \/{-, •)h is the induced norm of the corresponding Hilbert Space. Note that F'^ is considered 
as a Banach space under the Euclidean norm. The linear operator W is called the Frechet derivative and 
is usually denoted by dT(fQ) : H — > F'^. Observe that this definition is valid not only for Hilbert spaces, 



*Note, that for real valued functions the W and the CW derivatives constitute a conjugate pair flemma l2.13p . Thus if the 
W derivative vanishes, then the CW derivative vanishes too. The converse is also true. 
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but for general Banach spaces too. However, since we are mainly interested at Hilbert spaces, we present 
the main ideas in that context. It can be proved that if such a linear continuous operator W can be found, 
then it is unique (i.e., the derivative is unique). In the special case where z^ = 1 (i.e., the operator T takes 
values on F) using the Riesz's representation theorem, we may replace the linear continuous operator W 
with an inner product. Therefore, the operator T : H ^ F is said to be Frechet differentiable at /o, iff 
there exists a w & H, such that 

Mh^O \\h\\H 



where (•, ■)h is the dot product of the Hilbert space H and || • \\h is the induced norm. The element w* is 
usually called the gradient of T at /o and it is denoted by w* = VT(/o). 

For a general vector valued operator T = (Ti, . . . ,T^)'^ : H -^ F'^, we may easily derive that iff T is 
differentiable at /o, then T^ is differentiable at /o, for all i = 1, 2, . . . , z/, and that 

/{h,vTiifor)H\ 

dTifo){h) = : . (30) 

\{h,\/TMor)Hj 

To prove this claim, consider that since T is differentiable, there exists a continuous linear operator W such 
that 

1-^ \\T{fo + h)-T{fo)-W{h)\\p. ^^^ 



for all 6 = 1, . . . , z^. Thus, 



for all t = 1, 2, z^. The Riesz's representation theorem dictates that since W^ is a continuous linear operator, 
there exists w^ € H, such that W^{h) = {h, Wi)h, for all t = 1, . . . , i^. This proves that T^ is differentiable at 
/o and that vol — V^t(/o)) thus equation ([30|) holds. The converse is proved similarly. 

The notion of Frechet differentiability may be extended to include also partial derivatives. Consider the 
operator T : H^^ —^ F defined on the Hilbert space Ff^ with corresponding inner product: 




ii,„ (Tdh^h) T(M-w,(ky 



{f,9)Hi^ = ^(A,5t)H, 



=1 



where / = (/i, /2, • • • f^) g = (91,92, ■ ■ ■ 9^)- T{f) is said to be Frechet differentiable at /g in respect with 
/t, iff there exists a w £ H, such that 

l.m n/o + W.)-TW-(W...>„^ 

IMh^o \\h\\H 



where [/i]t = (0, 0, . . . , 0, /i, 0, . . . , 0)-^, is the element of iJ^ with zero entries everywhere, except at place t. 
The element w* is called the gradient of T at /q in respect with /^ and it is denoted by w* = VtT(/Q). The 
Frechet partial derivative at /q in respect with /^ is denoted by ^{fo), §f-{fo){h) = {h,w)M- 

It is also possible to define Frechet derivatives of higher order and a corresponding Taylor's series ex- 
pansion. In this context the n-th Frechet derivative of T at /q, i.e., d"T(/Q), is a multilinear map. If T has 
Frechet derivatives of any order, it can be expanded as a Taylor series, i.e., 

00 ^ 
T{fo + h) = J2 -d^T{fo){h, h,...,h). (32) 

n=0 

In relative literature the term d'^T{c){h,h, . . . ,h) is often replaced by d^T{c) ■ /i", which it denotes that 
the multilinear map d'^T{c) is applied to (h, h, . . . , h). 
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3.2 Complex Hilbert spaces 

Let 'H be a real Hilbert space with inner product (•, •)-^. It is easy to verify that TH? = % xli. is also a real 
Hilbert space with inner product 

{f.g)-H^ = {uf,ug)n + {vf,vg)n, (33) 

for / = {uf,Vf)'^, g = {ug,Vg)'^. Our objective is to enrich Ti^ with a complex structure. To this end, we 
define the space M = {f = u + iv, u, u € V.} equipped with the complex inner product: 

{f,9)m = {Uf,Ug)'H + {Vf,Vg)H+i{{Vf,Ug)n- {Uf,Vg)H), (34) 

for f = Uf + ivf, g = Ug + ivg. It may be easily proved that EI is a complex Hilbert space. In the following, 
this complex structure will be used to derive derivatives similar to the ones obtained from Wirtinger's 
calculus on C^. 

Consider the function T : A C M — )• C, T{f) = T{uf + ivf) = Ti{uf,Vf) + T2{uf, Vf)i, where Uf, Vf & V. 
and Ti, T2 are real valued functions defined on Ti^. Any such function, T, may be regarded as defined either 
on a subset of EI or on a subset of 7i^. Furthermore, T may be regarded either as a complex valued function, 
or as a vector valued function, which takes values in M?. Therefore, we may equivalently write: 

T{f) = T{uf + ivf) = Ti{uf,Vf) + T2{uf,Vf)i = {Ti{uf,Vf),T2iuf,Vf)f . 

In the following, we will often change the notation according to the specific problem and consider any 
element of / € EI defined either as f = Uf + ivf G EI, or as / = {uf,Vf) G Ti'^. In a similar manner, 
any complex number may be regarded as either an element of C, or as an element of M?. We say that T is 
Frechet complex dijferentiable at c G EI if there exists to G EI such that: 

,. T{c + h)-Tic)-{h,w)M „ 
imi 7—— = (J. 

Then w* is called the complex gradient of T at c and it is denoted as w* = \/T{c). The Frechet complex 

derivative of T at c is denoted as dT{c)[h) = {h,w)fi. This definition, although similar with the typical 

Frechet derivative , exploits the complex structure of EI. More specifically, the complex inner product that 

appears in the definition forces a great deal of structure on T. Similarly to the case of simple complex 

functions, from this simple fact follow all the important strong properties of the complex derivative. For 

example, it can be proved that if dT{c) exists, then so does d^T{c), for n G N. If T is differentiable at any 

c G A, T is called Frechet holomorphic in A, or Frechet complex analytic in A, in the sense that it can be 

expanded as a Taylor series, i.e., 

00 ^ 

r(c + /i) = V— d"T(c)(Ai,/i,...,^). (35) 

^-^ n\ 

n=0 

The proof of this statement is out of the scope of this manuscript. The expression "T is Frechet complex 
analytic at c" means that T is Frechet complex analytic at a neighborhood around c. We will say that T 
is Frechet real analytic, when both Ti and T2 have a Taylor's series expansion in the real sense. 

3.3 Cauchy-Riemann Conditions 

We begin our study, exploring the relations between the complex Frechet derivative and the real Frechet 
derivatives. In the following we will say that T is Frechet differentiable in the complex sense, if the complex 
derivative exists, and that T is Frechet differentiable in the real sense, if its real Frechet derivative exists 
(i.e., T is regarded as a vector valued operator T : Ti^ — )• T-L^). 

Lemma 3.1. Consider the Hilbert space M and a, 6 G EI. The limit 

{h*,a)M- {h,b)m „ .„„. 

I|/i||b^o \\h\\M 

if and only if a = b = 0. 
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Proof. Evidently, if a = 6 = 0, this limit exists and is equal to zero. For the converse, consider the case 
where h = t + iO, t & V.. Then equation (PUJ) transforms to lim||j||^^o '|u"jj = 0, which leads to a — b = 0. 

Similarly, if /i = + it, t G "H, then liniiuii .n ' lun = and thus a + b = 0. We conclude that 

ll''ll ri ^^ II'-IIK 

a = 6 = 0. D 



Proposition 3.2. Let T : A C HI ^ C 6e an operator such that T{f) = T{uf + ivf) = T{uf,Vf) = 
Ti(uf,Vf) + iT2{uf,Vf). If the Frechet complex derivative of T at a point c G A (i.e., dT(c) : H — > Cj 
exists, then Ti and T2 are differentiahle at the point c = (ci,ci) = ci + ic2, where ci,C2 € %. Furthermore, 

ViTi(ci,C2) = V2T2(ci,C2) and V2ri(ci,C2) = -Vir2(ci,C2). (37) 



Proof. Considering the first order Taylor expansion of T around c, we take: 

T{c + h)= T{c) + dT{c){h) + o{\\hU) = T{c) + {h, VT(c)*)h + oHMu), (38) 

where the notation o means that o(||/i||e)/||^||H —^ 0, as ||ft.||H —^ 0. Substituting \/T{c) = a + bi and 
h = hi + ih2, a, b, /ii, /12 £ "H, we have: 

T{c + h) =T{c) + {hi +ih2,a- ib)M + o(||/i||h) 

=Ti(ci,C2) + (/ii,a)H - {h2,b)M + ^[o{\\h\\u)] +^(^2(01,02) + (/i2,a)H + (/ii, &)h + 9[o(||^||h)]) • 

Therefore, 

Ti{ci + /ii,C2 + /i2) =Ti(ci,C2) + {hi,a)M - {h2,b)M + ^[o{\\h\\M)], (39) 

T2(ci + /ii, C2 + /i2) =T2(ci, C2) + (/i2, a)H + (/ii, % + 9[o(||/i||h)]. (40) 

Since o(||/i||h)/||^||h -^ 0, we also have 

K[o(|(/ii,/i2)|)]/||(/ii,/i2)||w2^0and9[o(|(/ii,/i2)|)]/||(/ii,/i2)k2^0as^^0. 

Thus, equations (J39ti40p are the first order Taylor expansions of Ti and T2 around (ci, C2). Hence we deduce 
that: 

ViTi(ci,C2) = a, V2ri(ci,C2) = -fo, Vir2(ci,C2) = b,S/2T2{ci,C2) =a. 

This completes the proof. D 

Equations ()44p are the Cauchy Riemann conditions with respect to the Prechet notion of differentiability. 
Similar to the simple case of complex valued functions, they provide a necessary and sufficient condition, for a 
complex operator T defined on H to be differentiahle in the complex sense, providing that T is differentiahle 
in the real sense. This is explored in the following proposition. 

Proposition 3.3. // the operator T : ^ C M ^ C, T(/) = ri(/) + iT2{f), where f = Uf+ivf, is Frechet 
differentiahle in the real sense at a point (ci,C2) G Ti^ and the Frechet Cauchy- Rieinann conditions hold: 

ViTi{ci,C2) = V2T2(ci,C2) andV2Ti{ci,C2) = -ViT2{ci,C2), (41) 

then T is differentiahle in the complex sense at the point c = (ci, C2) = ci + C2i G H. 
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Proof. Consider the first order Taylor expansions of Ti and T2 at c = ci + ic2 = (ci,C2) : 
Ti{c + h)= Ti(c) + {hi, ViTiic))^ + (/i2, V2ri(c))„ + 0(11^11^2), 

T2{C +h)= T2{C) + (/li, Vir2(c))^ + (/l2, V2r2(c))„ + o{\\h\\y2). 

Multiplying the second relation with i and adding it to the first one, we take: 

T{c + h)= T{c) + (/ii, Viri(c))^ + (/i2, V2ri(c))^ + i {hi,ViT2{c))^ + i {h2, V2T2{c))^ + 0(11^11^2) 
= T{c) + {hi, ViTiic) - iViT2{c))^ + (/i2, V2Ti(c) - iV2T2{c))^ + oHMm). 

To simplify the notation we may define 

ViT(c) = Viri(c) + iViT2(c) and V2T(c) = V2Ti(c) + iV2T2ic) 

and obtain: 

T{c + h)= T{c) + {hi, (ViT(c))*)e + (/i2, (V2T(c))*)jj + o{\\h\\^2). 

Next, we substitute hi and /i2 using the relations /ii = — ^^ — and /i2 = — g^— and use the sesquilinear 
property of the inner product of H: 

T{c + h) = T{c) + \ Ih, (Vir(c))* - i (V2T(c))A + \ lh\ (ViT(c))* + i (V2T(c))*\ + o(||/.|1^2) 

= T{c) + i ^f., (viT(c) + ^V2T(c)y ^ + ^ (^*' (^i^(^) - 7^2T(c)y ^ + o(||/.||h) 

= T[c) + i (^, (ViT(c) - iV2T(c))*)H + i (^*, (ViT(c) + iV2T(c))*)H + o(||fi||H). (42) 



It will be shown that equation (j42p is essential for the development of Wirtinger's calculus. To complete 
the proof of the proposition we compute the fraction that appears in the definition of the complex Frechet 
derivative: 

T{c + h)- T{c) - {h, w)m ^ ^ {h, (ViT(c) - iV2T{c))*)^ + 1 {h*, jViTjc) + iV2T{c)y)^ - {h, w)m 

II^IIh II^IIh 

o(II^IIh) 



Recall that, since o(||/i||h)/||^||h — >■ as ||/j-||h — ^ 0, for this limit to exist and vanish, it is necessary that 
ViT(c) + iV2T(c) = and w* = ViT(c)(c) — iV2T{c) (see lemma [3Tj. However, according to our 
definition, 

ViT(c) + iV2T(c) = (Viri(c) - V2r2(c)) + i {ViT2ic) + V2ri(c)) . 

Thus, T is differentiable in the Frechet complex sense, iff the Cauchy-Riemann conditions hold. Moreover, 
in this case: 

VT(c)=^(Vir(c)-iV2T(c)) 

=^ (ViTi(c) + tViT2{c)) - '- {V2Ti{c) + iV2T2(c)) 

=^ (ViTi(c) + V2T2(c)) + '- {ViT2{c) - V2Ti(c)) 
=ViTi{c)+iViT2{c) 

=V2T2{c)-iV2Ti{c). 

n 
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3.4 Prechet conjugate-complex Derivative 

An alternative definition of a complex derivative based on the Frechet notion of the differentiability on the 
Hilbert space M is the following. Consider an operator T : A C M ^ C, such that T{f) = T{uf + ivf) = 
T{uf,Vf) = Ti{uf,Vf) + iT2{uf,Vf), Uf,Vf G Ti. We shall say that T is Frechet conjugate-complex 
differentiable (or that it is differentiable in the Frechet conjugate- complex sense) at c G H, if there is a 
10 G H, such that the limit 

ll'illH-i'O II"IIh 

The continuous linear operator d^:T{c) : H — )• C, such that d.^T[c){h) = {h*,w)^ is called the Frechet 
conjugate- complex derivative of T at c and the element V^:T{c) = u;* G HI is called the Frechet conjugate- 
complex gradient of T at c. Following a procedure similar to the one presented in section 13.31 we may prove 
the following 

Proposition 3.4. Let T : A C M ^ C be an operator, such that T{f) = T{uf + ivf) = T{uf,Vf) = 
Ti{uf,Vf) + iT2{uf,Vf). If the Frechet conjugate- complex derivative of T at a point c ^ A (i.e., d^T[c) : 
H — )■ CJ exists, then Ti and T2 are differentiable at the point c = (ci,ci) = ci + ic2, where ci,C2 G 7i. 
Furthermore, 

ViTi(ci,C2) = -V2T2(ci,C2) and V2Ti(ci,C2) = ViT2{ci,C2). (44) 

These are called the Frechet conjugate Cauchy-Riemann conditions. 

Proposition 3.5. // the operator T : ACM^C, T{f) = Ti{f) + iT2{f), where f = Uf+ivf, is Frechet 
differentiable in the real sense at a point (ci, C2) G l-L^ and the Frechet conjugate Cauchy-Riemann conditions 
hold: 

Viri(ci,C2) = -V2r2(ci,C2) andV2Ti{ci,C2) = ViT2{ci,C2), (45) 

then T is differentiable in the Frechet conjugate- complex sense at the point c= (ci, C2) = ci + C2i G H. 

If an operator T is differentiable in the Frechet conjugate-complex sense, at every point of an open set 
A, we will say the T is Frechet conjugate holomorphic on A. It can be shown, that then T has a form of a 
Taylor series expansion around c G j4, i.e., 

00 ^ 
T{c + h) = y2 -,d!lT{c){h\h\ ..., h*). (46) 

n=0 

In this case, we will say that T is conjugate complex analytic at c. Note, that if T(/) is complex analytic 
at c, then T{f)* is conjugate -complex analytic at c. 

3.5 Frechet Wirtinger Derivatives 

It is evident, that if neither the Frechet Cauchy Riemann conditions, nor the Frechet conjugate Cauchy- 
Riemann conditions are satisfied for an operator T, then the Frechet complex derivatives cannot be exploited 
and the function cannot be expressed in terms of h or h*, as in the case of Frechet complex or conjugate- 
complex differentiable functions. Nevertheless, if T is Frechet differentiable in the real sense (i.e., Ti and T2 
are Frechet differentiable), we may still find a form of Taylor's series expansion. Recall, for example, that 
in the proof of proposition 13. 3| we concluded, based on the first order Taylor's series expansion of Ti, T2, 
that (equation 



T{c + h)= T{c) + \ {h, (ViT(c) - iV2T(c))*)H + ^{h\ (ViT(c) + iV2T(c))*)H + o(||^|| 
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One may notice that in the more general case, where T is Frechet real-differentiable, it's Taylor's expansion 
is casted in terms of both h and h*. This can be generalized for higher order Taylor's expansion formulas 
following the same rationale. Observe also that, if T is Frechet complex, or conjugate-complex differentiable, 
this relation degenerates (due to the Cauchy Riemann conditions) to the respective Taylor's expansion 
formula (i.e., ()35p or ()46p ). In this context, the following definitions come naturally. 
We define the Frechet Wirtinger's gradient (or W-gradient for short) of T at c as 

VfT{c) = \ {ViT{c) - iV2T(c)) = \ (ViUc) + V2r2(c)) + '- iViT^ic) - V2UC)) , (47) 

and the Frechet Wirtinger's derivative (or W -derivative) as ^(c) : IHI — t- C, such that ^{c){h) = 
{h,'V fT{c)*)^. Consequently, the Frechet conjugate Wirtinger's gradient (or CW-gradient for short) and 
the Frechet conjugate Wirtinger's derivative (or CW- derivative) of T at c are defined by: 

Vf*T{c) = i (ViT(c) + iV2T(c)) = ^ (ViTi(c) - V2T2(c)) + '- {ViT2{c) + V2Ti(c)) , (48) 

and ^(c) : M ^ C, such that ^{c){h) = (V/i, (V/*T(c))*)h. Note, that both the W-derivative and 
the CW-derivative exist, if T is Frechet differentiable in the real sense. In view of these new definitions, 
equation ([l2|l may now be recasted as follows 

T{c + h)= T{c) + {h, {VfT{c))X + {h*, {Vf*T{c))% + o{\\hy). (49) 

At first glance the definitions the W and CW derivatives seem rather obscure. Although, it is evident 
that they are defined so that that they are consistent with the Taylor's formula (equation (|42p). their 
computation seems quite difficult. However, this is not the case. We will show that they may be computed 
quickly using simple differentiation rules. First, observe that if T satisfies the Frechet Cauchy Riemann 
conditions then the W-derivative degenerates to the standard complex derivative. The following theorem 
establishes the fundamental property of W and CW derivatives. Its proof is rather obvious. 

Theorem 3.6. IfT is Frechet complex differentiable at c, then its W derivative degenerates to the standard 
Frechet complex derivative, while its CW derivative vanishes, i.e., 

VfT{c) = VT{c), VrT{c) = 0. 

Consequently, if T is Frechet conjugate- complex differentiable at c, then its CW derivative degenerates to 
the standard Frechet conjugate- complex derivative, while its W derivative vanishes, i.e., 

Vf*T{c) = V*T{c), VfT{c) = 0. 

In the sequel, we will develop the main differentiation rules of Frechet Wirtinger's derivatives. Most of 
the proofs of the following properties are straightforward. Nevertheless, we present them all for completeness. 

Proposition 3.7. IfT is Frechet differentiable in the real sense at c, then 

{VfTic))* = Vf^T*{c). (50) 



Proof. 



iVfT{c))* = i (Viri(c) + V2r2(c)) - '- (Vir2(c) - V2ri(c)) 

= i (Viri(c) - V2(-r2)(c)) + '- (Vi(-r2)(c) + v^nic)) 
= i^^rT^c)) . 

n 
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Proposition 3.8. IfT is Frechet differentiable in the real sense at c, then 

{Vf^T{c)Y = VfT*{c)). (51) 



Proof. 



[VrTic))* = i (Viri(c) - V2r2(c)) - '- (v^Uc) + V2ri(c)) 

= i (Viri(c) + V2(-r2)(c)) + ^ (Vi(-r2)(c) - V2Ti(c)) 

= (V/T*(c)). 



n 



Proposition 3.9 (Linearity). IfT,S are Frechet differentiable in the real sense at c and a,/3 G C, then 



Vf{aT + /35)(c) = aVfT{c) + /3V/S(c), (52) 

Vf*{aT + pS){c) = aVf*T{c) + (3Vf*S{c) (53) 



Proof. Let T(/) = T{uf + ivf) = T{uf,Vf) = Ti{uf,Vf) +iT2{uf,Vf), S{f) = S{uf + ivf) = S{uf,Vf) = 
Si{uf,Vf) + iS2{uf,Vf) be two complex functions and a,/3 G C, such that a = ai + ia2, /3 = /3i + i/32. Then 

R{f) =aT{f) + /35(/) = (ai + ia2){Ti{uf,Vf) + iT2{uf,Vf)) + (/3i + iP2){Siiuf, Vf) + iS2iuf, Vf)) 
= {aiTi{uf,Vf) -a2T2{uf,Vf) + f3iSi{uf,Vf) - j32S2{uf,Vf)) 
+ i{aiT2{uf,Vf) + a2Ti{uf,Vf) + PiS2{uf,Vf) + P2Si{uf,Vf)) . 

Thus, the Frechet W-derivative of Ft will be given by: 

VfR{c) =i (Vii?i(c) + V2i?2(c)) + '- (Vii?2(c) - V2i?i(c)) 

=i (aiViri(c) - a2ViT2{c) + /3iViSi(c) - /32ViS2(c) 
+aiV2r2(c) + a2V2ri(c) + /3iV252(c) + /32V2Si(c)) 
+ \ (aiVir2(c) + a2Viri(c) + /3iVi52(c) + /32Vi5i(c) 

-aiV2ri(c) + Q2V2r2(c) - PlV2Si{c) + P2^2S2{C)) 

=\{ai + ia2)ViTi{c) + ^(ai + ia2)ViT2{c) + i(/3i + i/32)ViSi(c) + ^(/3i + i/32)Vi52(c) 

+ ^("i + ^a2)V2r2(c) - '-{ai + ia2)V2Ti(c) + 2 (/^i + ^/32)V252(c) - ^(A + i/52)V25i(c) 
=a Q (ViTi(c) + V2T2(c)) + ^ (Vir2(c) - V2Ti(c)) 

+ /5 (^ (Vi5i(c) + V2S2(C)) + '- (ViS2(c) - V25i(c)) 
=aVfT{c) + /3VfS{c). 
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On the other hand, in view of Propositions [3^ and [3771 and the hnearity property of the Frechet W-derivative, 
the Frechet CW-derivative of -R at c will be given by: 

VrR{c) =V^.(«T + /?S)(c) = {Vf{aT + f3Snc)r 

= {Vf{a*T* + (3*S*){c))* = {a*VfT*{c) + /3*V_f S*(c))* 
=a {VfT*{c))* + /3 {VfS*{c))* = aVf*T{c) + /3Vj.5(c). 

D 



Proposition 3.10 (Product Rule). IfT,S are Frechet dijjerentiable in the real sense ai c G H, then 

Vf{T ■ S){c) = VfT{c)S{c) + T{c)VfS{c), (54) 

Vf*{T ■ S){c) = Vf*Tic)S{c)+T{c)Vf*Sic). (55) 



Proof. Let T(/) =T{uf + ivf) = T{uf,Vf) = Ti{uf,Vf) +iT2{uf,Vf), S{f) = S{uf + ivf) = S{uf,Vf) = 
Si{uf,Vf) + iS2{uf,Vf), be two complex functions Frechet differentiable at c. Consider the complex-valued 
operator R defined as R{f) = T{f)S{f). Then 

R{f) = (Ti(/) + ir2(/))(5i(/) + iS^if)) = (ri(/)5i(/) - T2{f)S2{f)) + i{Ti{f)S2{f) + T2{f)Siif)). 

Hence the Frechet W-derivative of i? at c is given by: 

VfR{c) =i (Vii?i(c) + V2i?2(c)) + '- (Vii?2(c) - V2i?i(c)) 

=i (ViiTiSi - T2S2){c) + V2{TiS2 + T2S{){c)) + '- {Vi{TiS2 + T2Si){c) - V2{TiSi - T2S2){c)) 
=i (Vi(Ti5i)(c) - Vi(T252)(c) + V2(Ti52)(c) + V2(T25i)(c)) 

+ '- (Vi(ri52)(c) + Vi(r2Si)(c) - V2(ri5i)(c) + V2(r2S2)(c)) 

Applying the chain rule of the ordinary Frechet calculus we take: 

VfR{c) =^ (Vi(ri)(c)Si(c) + ViSi(c)Ti(c) - ViT2(c)52(c) - ViS2{c)T2{c) 
+V2Ti{c)S2{c) + \/2S2{c)Ti{c) + V2T2{c)Si{c) + V2Si{c)T2{c)) 
+ '- {ViTi{c)S2{c) + ViS2(c)Ti(c) + ViT2{c)Si{c) + ViSi{c)T2{c) 
-V2Ti{c)Si{c) - V2Si{c)Ti{c) + V2T2(C)52(C) + V2S2{C)T2{C)) . 

After factorization we obtain: 

VfR{c) =Si{c) Q (Viri(c) + V2r2(c)) + '- (Vir2(c) - V2r2(c))) 
+ S2{c) Q (-Vir2(c) + V2ri(c)) + '- (ViTiic) + V2r2(c)) 

+ Ti{c) Q (ViSi(c) + V252(C)) + '- (Vi52(c) - V2Si(c)) 

+ T2{c) Q (-Vi52(c) + V25i(c)) + '- (Vi5i(c) + V2S2{C)) 
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Considering that l/i = —i, we take: 

VfRic) =Si{c) Q (Viri(c) + V2r2(c)) + '- (ViTsCc) - V2ri(c))) 

+ iS2{c) Q (Viri(c) + V2r2(c)) + '- {ViT2{c) - V2Ti(c)) 
+ Ti{c) Q (ViSi(c) + V252(c)) + '- (Vi52(c) - V2Si(c)) 
+ iT2{c) Q (Vi5i(c) + V252(C)) + '- (ViS2(c) - V25i(c)) 

={Si{c) + iS2{c))\/fT{c) + {Ti{c) + iT2{c))VfS{c), 

which gives the result. 

The product rule of the Frechet CW-derivative follows from the product rule of the W-derivative and 
Propositions 13.71 13.71 as follows: 

Vj*(TS)(c) = {Vf{TSnc)r = {Vf{T*S*){c)r 
= {VfT*{c)S*{c) + VfS*{c)T*{c))* 
= Vf^T{c)S{c) + Vf*S{c)T{c). 

D 
Lemma 3.11 (Reciprocal Rule). IfT is Frechet dijjerentiable in the real sense at c and T{c) ^ 0, then 



V. (i) (c) 



T\c) ^ 
1 \ _ Vf*T{c) 



(56) 



Proof. Let T{f) = T{uf + ivf) = T{uf,Vf) = Ti{uf,Vf) + iT2{uf,Vf) be a complex function, Frechet 
differentiable in the real sense at c, such that T(c) 7^ 0. Consider the function R{f) = 1/T(/). Then 



T?{f)+Ti{f) Tf{f)+Tiify 
For the partial derivatives of Ri, R2 we have: 

Viri(c)(T2(c) + r|(c)) - 2r2(c)Viri(c) - 2ri(c)r2(c)Vir2(c) 



Vii?i(c) 

V2i?l(c) 
Vli?2(c) 
V2i?2(c) 



(r2(c)+r|(c))2 
V2ri(c)(r2(c) + r|(c)) - 2rf (c)V2Ti(c) - 2ri(c)r2(c)V2r2(c) 

(r2(c)+r|(c))2 

ViT2(c)(rf (c) +T|(c)) - 2r|(c)ViT2(c) - 2Ti{c)T2{c)V iTijc) 

{Tf{c)+Ti{c)Y 
V2T2(c)(rf (c) + T|(c)) - 2T|(c)V2T2(c) - 2ri(c)T2(c)V2ri(c) 

(T2(c) + r|(c))2 
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Therefore, 

VfR{c) =^ (Viiii(c) + V2i?2(c)) + '- {ViR2{c) - V2Ri{c)) 

^ (Viri(c) (-r2(c) + Ti{c) + 2iTi{c)T2{c)) 



+ViT2{c) (-2Ti(c)r2(c) - iTl{c) + iTiic)) 

+V2T2{c) {-Tfic) + r|(c) + 2iTi{c)T2{c)) + V2Ti(c) (2ri(c)r2(c) + iTf{c) - iT^{c))) 

T?ic)-Ti{c)-2zT,{c)T2{c) 



2(r2(c) + r|(c))2 
VfT{c) 



(- (Viri(c) + V2r2(c)) - i (Vir2(c) - V2ri(c))) 



T^(c) 



To prove the corresponding rule of the Frechet CW-derivative we apply the reciprocal rule of the Frechet 
W-derivative as well as Propositions 13.71 13.81 



iT*{c)rJ {T{c)Y 

D 



Proposition 3.12 (Division Rule). IfT,S are Frechet dijjerentiable in the real sense at c and S{c) ^ 0, 
then 

V,(|),e,^M>mzZ(£)ZiM, (58) 

V,. (I) ,e, . V,.T(c).(c)-T(c)V,.S(c) ^ ^^^^ 

Proof. It follows immediately from the multiplication rule and the reciprocal rule ( ^1^ = T{c) ■ -^t^ I . D 

Proposition 3.13 (Chain Rule). Consider the functions T : H ^ C and S : C —^ C so that they are 
differentiable in the real sense at c and zq = T[c) respectively. Then the operator R = S oT is differentiahle 
in the real sense at c, and 

VfS o T{c) = —{T{c))VfT{c) + —iTic))Vf{T*){c), (60) 

Vf^So T{c) = ^iT{c))VrT{c) + ^{T{c))Vf^{T*){c). (61) 

Proof Consider the function R{f) = S o T(/) = UR{f) + ivR{f) = us{Ti{f),T2{f)) + ivs{Ti{f),T2{f)). 
Then the Frechet partial derivatives of Ri and R2 are given by the chain rule: 

Vii?i(c) = ^(T(c))Viri(c) + ^(T(c))Vir2(c), 

V2i?l(c) = ^(r(c))V2Ti(c) + ^(T(c))V2r2(c), 

Vii?2(c) = ^(T(c))Viri(c) + ^(T(c))Vir2(c), 
V2i?2(c) = ^(T(c))V2Ti(c) + ^(T(c))V2r2(c). 
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In addition, we have: 

^(T(c))V^r(c) =\ {^^{T{c))V,T,{c) + ^-^{T{c))V,T,{c) + ^^{T{c))V,T,{c) - t^{T{c))V,T,{o) 

+ ^{T{c))V,T,{c) + ^{T{c))V,Uc) + i^(T(c))Vir2(c) - ^^(T{c))V2T,{c) 
ay oy ay ay 

+ .^(T(c))ViTi(c) + ^^{T{c))V2T2{c) - ^-^{T{c))V,T,{c) + ^{T{c))V2T,{c) 
ox ox ox ox 

-i^{T{c))V,T,{c) - ^^{T{c))V2Uc) + ^{T{c))VrT,{c) - ^{T{c))V2T,{c) 
and 
||(T(c))V^T*(c) =\ {^{T{c))V^T,{c) - ^{T{c))V,Uc) - ^^-^{T{c))V^Uc) - ^^-^{T{c))V,T,{c) 

- ^{T{c))V^n{c) + ^(T(c))V2T2(c) + ^^(T(c))Vir2(c) +.^(T(c))V2Ti(c) 
+ i^(T(c))Viri(c) - ^^(T(c))V2^2(c) + ^{T{c))V^Uc) + ^(T(c))V2ri(c) 
+i^(T(c))ViTi(c) - ^^[T{c))V2T2{c) + ^-^{T{c))V,T,{c) + ^{T{c))V2T,{c 

Summing up the last two relations and eliminating the opposite terms, we obtain: 

^iTic))VfT{c) + ^{Tic))VfT*{c) J- (^-^{T{c))V^T,{c) - t^-^{T{c))V,T,{c) + ^{T{c))V,T,{c) 

+ i^{T{c))ViT2{c) + i^(T(c))Viri(c) + ^(T(c))V2ri(c) 
oy ox ox 

-i^{T{c))V2Uc) + ^{T{c))V^Uc) 

= ^ (Vii?i(c) + V2i?2(c)) + ^ (Vifi2(c) - Vi/?i(c)) 

=VfR{c). 

To prove the chain rule of the Frechet CW-derivative, we apply the chain rule of the W-derivative as 
well as Propositions 13.71 13-81 and obtain: 

VrR{c) = {VfR*{c)r = i^-^(T{c))VfT{c) + ^T{c))VfT*{c)^ 
-(T(c))j (VjT(c))*+(^^-^(T(c))J {VfT*{c)r 



dS* 



dz 

-iT{c))VrT*{c) + —iTic))Vf*T{c) 



dz* dz 

which completes the proof. D 

The following rules may be immediately proved using the definition of Frechet W and CW derivatives 
and the aforementioned rules. 

1. If T(/) = (/, w)m, then V/T = w*, V f^T = 0. 

2. If T(/) = {w, /)h, then V fT = 0, Vj.T = w. 

3. If T(/) = (r , w)u, then V/T = 0, V f*T = w* . 

4. If T{f) = {w, /*)h, then VjT = w, S/f^T = 0. 
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3.6 Generalized Wirtinger's calculus applied on real valued functions 

In non-linear complex signal processing, we are often interested in minimization problems of real valued 
cost functions defined on certain complex Hilbert spaces. Therefore, in order to successfully implement the 
associated minimization algorithms, the gradients of the respective cost functions need to be deployed. We 
may compute the gradients, either by employing ordinary Frechet calculus, that is regarding the complex 
Hilbert space as a cartesian product of real Hilbert spaces, or by using Wirtinger's calculus. Both cases 
will eventually lead to the same results, but the application of Wirtinger's calculus provides a more elegant 
and comfortable alternative, especially if the cost function, by its definition, is given in terms of / and /* 
(where / is an element of the respective Hilbert space). 

As the function under consideration T{f) is real valued, the Frechet W and CW derivatives are simplified, 
i.e., 

VfT{c) = i (Vir(c) - iV2T{c)) and Vf^T{c) = ^ (Vir(c) + iV2r(c)) 
and the following important property can be derived. 

Lemma 3.14. If f : A QM—^M is Frechet dijferentiable in the real sense, then 

{VfTic)r = Vf^T{c). (62) 

An important consequence is that if T is a real valued function defined on H, then its first order Taylor's 
expansion at c is given by: 

T{c + h) = T{c) + {h, {VfT{c))X + {h\ {VrT{c))X + o(||^||h) 
= T{c) + {h, {VfTic))*)^ + {{h, V^.r(c)y * + oiMn) 
= Tic) + ^[{h,iVfTic)r)^] + oi\\h\\n). 

However, in view of the Cauchy Riemann inequality we have: 

^[{h,iVfT{c))X]<\{h,{VfTic)r)^\ 
<\\h\\M\\Vf*T{c)\\^. 

The equality in the above relationship holds, if /i f| V f*T[c). Hence, the direction of increase of T is 
V f*T{c). Therefore, any gradient descent based algorithm minimizing T(/) is based on the update scheme: 

/n = /n-l-^-V^.r(/„„i). (63) 

Assuming differentiability of T, a standard result from Frechet real calculus states that a necessary 
condition for a point c to be an optimum (in the sense that T{f) is minimized) is that this point is a 
stationary point of T, i.e. the partial derivatives of T at c vanish. In the context of Wirtinger's calculus we 
have the following obvious corresponding result. 

Proposition 3.15. If T : X QM. ^ C is Frechet differentiable at c in the real sense, then a necessary 
condition for a point c to be a local optimum (in the sense that T{c) is minimized or maximized) is that 
either the Frechet W, or the CW derivative vanished. 



^Note, that for real valued functions the W and the CW derivatives constitute a conjugate pair flemma l3.14p . Thus if the 
W derivative vanishes, then the CW derivative vanishes too. The converse is also true. 
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